Gaussian processes (GPs) provide a powerful non-parametric framework for reasoning over functions. Despite appealing theory, its superlinear computational and memory complexities have presented a long-standing challenge. State-of-the-art sparse variational inference methods trade modeling accuracy against complexity. However, the complexities of these methods still scale superlinearly in the number of basis functions, implying that that sparse GP methods are able to learn from large datasets only when a small model is used. Recently, a decoupled approach was proposed that removes the unnecessary coupling between the complexities of modeling the mean and the covariance functions of a GP.

I agree the standard error can be misleading due to the correlations induced by the datasets while the standard error of the ranks is more informative. After the discussion with other reviewers, I think adding the motivation (or derivation) for Equation 9 will definitely make the paper stronger and I encourage the authors to do so in their final version. This paper proposes a novel RKHS parameterization of decoupled GPs that admits efficient natural gradient computation. Specifically, they decompose the mean parameterization into a part that shares the basis with the covariances, and an orthogonal part that models the residues that the standard decoupled GP (Chen and Boots, 2017) fails to capture. This construction allows for a straightforward natural gradient update rule.

Gaussian processes (GPs) provide a powerful non-parametric framework for reasoning over functions. Despite appealing theory, its superlinear computational and memory complexities have presented a long-standing challenge. State-of-the-art sparse variational inference methods trade modeling accuracy against complexity. However, the complexities of these methods still scale superlinearly in the number of basis functions, implying that that sparse GP methods are able to learn from large datasets only when a small model is used. Recently, a decoupled approach was proposed that removes the unnecessary coupling between the complexities of modeling the mean and the covariance functions of a GP. It achieves a linear complexity in the number of mean parameters, so an expressive posterior mean function can be modeled.